Unwrinkling systems and methods

ABSTRACT

Due to rapid advancement in computing technology of both hardware and software, the labor intensive sewing process has been transformed into a technology-intensive automated process. During an automated process, product materials may become wrinkled or folded, which can slow down the automated process or result in human intervention. Various examples are provided related to the automation of sewing robots, and removal of wrinkles from products. Images of material on a work table can be captured and analyzed to determine if there are wrinkles or folds in the product. The robot can remove the wrinkle or fold by manipulating the material through the use of end effectors such as, e.g., budgers.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part claiming priority to, and thebenefit of, co-pending U.S. non-provisional application entitled“Unwrinkling Systems and Methods” having Ser. No. 16/704,578, filed Dec.5, 2019, the entirety of which is hereby incorporated by reference.

FIELD OF THE DISCLOSURE

The present disclosure is generally related to the automation of sewingusing sewing robots. More specifically, the disclosure is related todealing with products that wrinkle during an automated sewing process.

BACKGROUND

Clothing is one of the necessities of human life and a means of personalexpression. As such, clothing or garment manufacturing is one of theoldest and largest industries in the world. However, unlike other massindustries such as the automobile industry, the apparel industry isprimarily supported by a manual production line. The need for automationin garment manufacturing has been recognized by many since the early1980s. During the 1980s, millions of dollars were spent on apparelindustry research in the United States, Japan and industrialized Europe.For example, a joint $55 million program between the Ministry ofInternational Trade and Industry (MITI) and Industry, called the TRAAS,was started in 1982. The goal of the program was to automate the garmentmanufacturing process from start, with a roll of product, to finish,with a complete, inspected garment. While the project claimed to besuccessful and did demonstrate a method to produce tailored women'sjackets, it failed to compete with traditional methodologies.

Currently, a sewing machine uses what is known as a feed dog to move theproduct through the sewing head relying on the operator to maintain theproduct orientation and keep up with the feed rate for the rest of thegood. Previous attempts at automated sewing used the feed dogs on astandard sewing machine and had a robot perform exactly the operations ahuman user would perform. Desirable in the art is an improved automatedsewing machine that would improve upon the conventional automated sewingdesigns that are unable to process wrinkles and/or folds, which may beintroduced while the product to be sewn is moving on a robotic sewingsystem. The value of this disclosure is to provide a flat piece of aproduct to be sewn for an automated sewing machine allowing the machineto function properly without mistakes caused by a wrinkle or bump in theproduct to be sewn.

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem mentioned in the backgroundsection or associated with the subject matter of the background sectionshould not be assumed to have been previously recognized in the priorart. The subject matter in the background section merely representsdifferent approaches, which in and of themselves may also correspond toimplementations of the claimed technology.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various examples of systems,methods, and embodiments of various other aspects of the disclosure. Anyperson with ordinary skills in the art will appreciate that theillustrated element boundaries (e.g., boxes, groups of boxes, or othershapes) in the figures represent one example of the boundaries. It maybe that in some examples one element may be designed as multipleelements or that multiple elements may be designed as one element. Insome examples, an element shown as an internal component of one elementmay be implemented as an external component in another, and vice versa.Furthermore, elements may not be drawn to scale. Non-limiting andnon-exhaustive descriptions are described with reference to thefollowing drawings. The components in the figures are not necessarily toscale, emphasis instead being placed upon illustrating principles.Moreover, in the drawings, like reference numerals designatecorresponding parts throughout the several views.

FIG. 1 illustrates an example of a wrinkle removal system, according tovarious embodiments of the present disclosure.

FIG. 2 illustrates an example of a base module, according to variousembodiments of the present disclosure.

FIG. 3 illustrates an example of a fabric detection module, according tovarious embodiments of the present disclosure.

FIG. 4 illustrates an example of a depth filtering module, according tovarious embodiments of the present disclosure.

FIG. 5 illustrates an example of a wrinkle removal module, according tovarious embodiments of the present disclosure.

FIG. 6 illustrates an example of a test module, according to variousembodiments of the present disclosure.

FIGS. 7A and 7B illustrate a wrinkle removal example, according tovarious embodiments of the present disclosure.

FIG. 8 illustrates an end effector example, according to variousembodiments of the present disclosure.

DETAILED DESCRIPTION

Disclosed herein are various examples related to automation of sewingusing sewing robots. Some embodiments of this disclosure, illustratingall its features, will now be discussed in detail. The words“comprising,” “having,” “containing,” and “including,” and other formsthereof, are intended to be equivalent in meaning and be open ended inthat an item or items following any one of these words is not meant tobe an exhaustive listing of such item or items, or meant to be limitedto only the listed item or items.

It must also be noted that as used herein and in the appended claims,the singular forms “a,” “an,” and “the” include plural references unlessthe context clearly dictates otherwise. Although any systems and methodssimilar or equivalent to those described herein can be used in thepractice or testing of embodiments of the present disclosure, thepreferred, systems, and methods are now described.

Embodiments of the present disclosure will be described more fullyhereinafter with reference to the accompanying drawings in which likenumerals represent like elements throughout the several figures, and inwhich example embodiments are shown. Embodiments of the claims may,however, be embodied in many different forms and should not be construedas limited to the embodiments set forth herein. The examples set forthherein are non-limiting examples and are merely examples among otherpossible examples.

Referring to FIG. 1, shown in an example of a wrinkle removal system. Asillustrated in the example of FIG. 1, the system can comprise a roboticsystem 102, which can include a processor 104, memory 106, an interfacesuch as, e.g., a human machine interface (HMI) 108, I/O device(s) 110,networking device(s) 112, a sewing device 114, fabric mover(s) 116,secondary operation device(s) 118, sensor(s) 120, and a local interface122. Sensor(s) 120 can include, but are not limited to, devices that candetect wrinkles using, e.g., electromagnetic field (EMF), ultrasound,physical touch, or other sensing technology. The sensor(s) 120 cancomprise a vision device or camera 124 such as, e.g., a RGB camera, anRGB-D camera, a near infrared (NIR) camera, stereoscopic camera,photometric stereo camera (single camera with multiple illuminationoptions), etc. The robotic system 102 can also include a base module126, a fabric detection module 128, a depth filtering module 130, awrinkle removal module 132, and/or a test module 134, which may beexecuted to implement wrinkle removal.

The robotic system 102 can move a piece of product across a surface of awork table and determine if the piece of product contains any wrinklesthrough the base module 126. If it is determined that the productcontains a wrinkle, through the depth filtering module 130, the wrinkleremoval module 132 can initiate fabric movers 116, or material movers,(e.g., budgers, fingers, air jets, etc.) to move in specific directionsto remove the wrinkle from the product. The test module 134 can thentest the product to determine if the wrinkle has been removed or isstill present. If the wrinkle is still present, the wrinkle removalmodule 132 can be initiated again, by the robotic system 102.

The processor 104 can be configured to decode and execute anyinstructions received from one or more other electronic devices orservers. The processor can include one or more general-purposeprocessors (e.g., INTEL® or Advanced Micro Devices® (AMD)microprocessors) and/or one or more special purpose processors (e.g.,digital signal processors or Xilinx® System on Chip (SOC) fieldprogrammable gate array (FPGA) processor). The processor 104 may beconfigured to execute one or more computer-readable programinstructions, such as program instructions to carry out any of thefunctions described in this description.

The Memory 106 can include, but is not limited to, fixed (hard) drives,magnetic tape, floppy diskettes, optical disks, Compact Disc Read-OnlyMemories (CD-ROMs), and magneto-optical disks, semiconductor memories,such as ROMs, Random Access Memories (RAMs), Programmable Read-OnlyMemories (PROMs), Erasable PROMs (EPROMs), Electrically Erasable PROMs(EEPROMs), flash memory, magnetic or optical cards, or other type ofmedia/machine-readable medium suitable for storing electronicinstructions. The Memory 106 can comprise modules that can beimplemented as a program executable by processor(s) 104.

The interface(s) or HMI 108 can either accept inputs from users orprovide outputs to the users or may perform both the actions. In onecase, a user can interact with the interfaces using one or moreuser-interactive objects and devices. The user-interactive objects anddevices may comprise user input buttons, switches, knobs, levers, keys,trackballs, touchpads, cameras, microphones, motion sensors, heatsensors, inertial sensors, touch sensors, or a combination of the above.Further, the interfaces can either be implemented as a command lineinterface (CLI), a human machine interface (HMI), a voice interface, ora web-based user-interface, at element 108.

The input/output devices or I/O devices 110 of the robotic system 102can comprise components used to facilitate connections of the processor104 to other devices such as, e.g., a knife device, sewing device 114,fabric (or material) mover(s) 116, secondary operation device(s) 118and/or sensor(s) 120 and therefore, for instance, can comprise one ormore serial, parallel, small system interface (SCSI), universal serialbus (USB), or IEEE 1394 (i.e. Firewire™) connection elements.

The networking device(s) 112 of the robotic system 102 can comprise thevarious components used to transmit and/or receive data over a network.The networking device(s) 112 can include a device that can communicateboth inputs and outputs, for instance, a modulator/demodulator (i.e.modem), a radio frequency (RF) or infrared (IR) transceiver, atelephonic interface, a bridge, a router, as well as a network card,etc.

The sewing device 114 of the robotic system 102 facilitates sewing theproduct materials together and can be configured to sew a perimeter oralong markings on the product material based on tracking a generatedpattern. In additional embodiments, the sewing device 114 can include aknife device in order to cut threads, stitches, materials from theworkpiece etc. The fabric mover(s) 116, or material mover(s), of therobotic system 102 can facilitate moving the product material(s) duringthe cutting and sewing operations, at element 116. The fabric, ormaterial, mover(s) 116 can comprise end effectors, pneumatics, edgecontrols, or other appropriate material manipulator(s). The secondaryoperation device(s) 118 can include stacking device(s), foldingdevice(s), label manipulation device(s), and/or other device(s) thatassist with the preparation, making and/or finishing of the sewnproduct.

The sensor(s) 120 of the robotic system 102 can facilitate detecting themovement of the product material(s) and inspecting the productmaterial(s) for defects and/or discrepancies during a sewing and cuttingoperation. Further, the vision device(s) 124 can facilitate detectingmarkings on the product before cutting or sewing the material. A sensor120 can comprise, but is not limited to, camera(s) and/or visiondevice(s) 124 such as, e.g., an RGB-D camera, near IR camera, time offlight camera, Internet protocol (IP) camera, light-field camera,monorail camera, multiplane camera, rapatronic camera, stereo camera,still camera, thermal imaging camera, acoustic camera, rangefindercamera, etc., at element 120. The RGB-D camera is a digital camera thatcan provide color (RGB) and depth information for pixels in an image.

The local interface 122 of the robotic system 102 can be, for example,but not limited to, one or more buses or other wired or wirelessconnections, as is known in the art. The local interface 122 can haveadditional elements, which are omitted for simplicity, such ascontrollers, buffers (caches), drivers, repeaters, and receivers, toenable communications. Further, the local interface 122 can includeaddress, control, and/or data connections to enable appropriatecommunications among the components, at element 122.

As shown in FIG. 1, the robotic system 102 includes a base module 126which can activate the fabric detection module 128, depth filteringmodule 130, wrinkle removal module 132, and test module 134, as will bediscussed. The fabric detection module 128 can determine if a piece ofproduct material has moved on the work table within the RGB-D camera's(or vision device 124) view. The depth filtering module 130 can analyzethe captured image(s) to determine if a wrinkle is present on theproduct material. If there is a wrinkle, this information is returned tothe wrinkle removal module 132, at element 130. The wrinkle removalmodule 132 can determine and activate the necessary budgers (or otherfabric or material mover(s) 116) on the work table to remove the wrinklefound by the depth filtering module 130 and initiates the test module134 to determine if there are any wrinkles remaining, at element 132.The test module 134 which is initiated by the wrinkle removal module132, performs a task similar to the depth filtering module 130 todetermine if there are any wrinkles remaining within the piece of theproduct. If there are no more wrinkles, then the process ends. However,if there are wrinkles remaining the test module 134 initiates thewrinkle removal module 132 to remove the remaining wrinkles, at element134.

An example of wrinkle removal is displayed by the two images of FIGS. 7Aand 7B. FIG. As will be discussed, 7A shows an image of a piece of theproduct material that contains a wrinkle and FIG. 7B shows the piece ofproduct that does not contain the wrinkle as will be discussed. Thewrinkle can be reduced or removed using fabric, or material, mover(s)116 such as end effectors, pneumatics, edge controls, or otherappropriate material manipulator(s). An end effector example is providedin FIG. 8, which displays a series of end effectors that may be used asopposed to or in combination with, e.g., the budger system disclosed inU.S. Pat. No. 8,997,670 (“Conveyance System that Transports Fabric”),which is hereby incorporated by reference in its entirety. A budger caninclude a steered ball that can rotate in two perpendicular axis.Traction between the product material and the ball can be enhanced by aslight vacuum drawing a flow of air through the product via a series ofholes in the ball. An array of small budgers can be used to transportthe product to the sewing device 114, and ensure that the productmaterial lays flat in the correct orientation and without wrinkles.

Pneumatics such as, e.g., directional air jets blowing perpendicular tothe wrinkle's primary (or longitudinal) axis can be used to pull thewrinkle out of a piece of product material. Blowers can be embedded inthe work table (or work surface). The separate blowers can supply acontrolled air flow through nozzles or orifices that provide directionalcontrol or can be configured to rotate to direct air flow. Inalternative embodiments, blowers can be suspended above the worksurfaceor attached to an industrial robot to direct air jets for smoothing thepiece of product material. In some embodiments, blowers may be mountedin a fixed location and the fabric or material mover(s) can shift thepiece of product material to appropriately orient it with respect to theblowers for removal.

The piece of product material on either side of the wrinkle can also bespread apart through mechanical actuation or manipulation. Two separateend effectors can be positioned on either side of the wrinkle and moveapart. In other embodiments, a single split end effector with actuationcan be positioned over the wrinkle and moved apart. In alternativeembodiments, a single belt or an array of belts on one or both sides ofthe wrinkle can be used to move the material. In some embodiments, theproduct material can be moved from underneath or above by driven rollersor wheels (e.g., budgers) that may or may not have directional control.In various embodiments, the wrinkle can be positioned over a conveyor orsimilar moving surface which can adjust the piece of product material.The conveyor or moving surface may or may not reciprocate or beflexible. For example, the surface can be configured to shift by aconstrained amount, e.g., 100 mm or less.

The piece of product material can be adjusted in one or more directionsto reduce or remove the wrinkle. The edge of the piece of productmaterial or near the edge of the material may be secured in place (e.g.,using a clamp, suction, etc.) at one or more locations appropriate tothe location of the wrinkle, or can be secured to an actuator or endeffector which may be moving the piece of product. The piece of productmaterial can then be adjusted to remove it. The adjustments can becarried out while the piece of product material is stationary or inmotion (e.g., during processing of the piece of product). The materialcan be pulled or manipulated as described. The adjustment can be appliedalong one or more edges to remove the wrinkle. The material may besecured (clamped) at an edge or near an edge on the other side of thewrinkle, or the piece of product material may be adjusted without beingsecured along an edge.

In various embodiments, a constant or intermittent tension can beapplied to the piece of product material in some or all directions toremove or prevent wrinkles. For instance, concentric rings of air jetspointing outward from a center point can tension a piece of productpositioned or moving over the air jets to remove or prevent wrinkles inthe material. In some implementations, adjustments can be made to removethe wrinkle during movement of the piece of product material by theactuators or end effectors. For example, budgers can apply tension to apiece of product while they are moving it on a work surface. A velocityvector pointing away from the center of the piece of product or from thewrinkle can be added to the budger's base motion to remove the wrinkle.Adjustments can also be applied during sewing or other processing of thepiece of product material.

Functioning of the base module 126 of the robotic system 102 will now beexplained with reference to FIG. 2. One skilled in the art willappreciate that, for this and other processes and methods disclosedherein, the functions performed in the processes and methods may beimplemented in differing order. Furthermore, the outlined steps andoperations are only provided as examples, and some of the steps andoperations may be optional, combined into fewer steps and operations, orexpanded into additional steps and operations without detracting fromthe essence of the disclosed embodiments.

The flow chart of FIG. 2 shows the architecture, functionality, andoperation of a possible implementation of the base module 126. Theprocess begins at 202 with the base module 126 initiating the fabricdetection module 128 of the robotic system 102, which captures one ormore (e.g., a series of) images using one or more sensor(s) 120 such as,e.g., the RGB-D camera in order to determine if a piece of a product hasbeen identified and returns to the base module 126. Next, the depthfiltering module 130 of the robotic system 102 is initiated at 204. Thedepth filtering module 130 analyzes the images captured at 202 from thefabric detection module 128 and determines if the product contains awrinkle and returns to the base module 126.

At 206, the wrinkle removal module 132 is initiated and determines whichfabric mover(s) 116 such as, e.g., one or more budger(s) should beinitiated in order to remove the identified wrinkle and activates thenecessary budgers (or other fabric mover(s) 116) to remove the wrinklefrom the product material, and then captures another series of imagesusing the sensor(s) 120 (e.g., the RGB-D camera) and returns to the basemodule 126. In some implementations, fabric mover(s) 116 to remove thewrinkle can be identified by the depth filtering module 130. Then at208, the test module 134 is initiated, which analyzes one or more newimages, or other sensor data, captured in the wrinkle removal module 132to determine if there are any wrinkles remaining (or unacceptable forprocessing the piece of the product. If there are wrinkles remaining at210, then the process returns to 206 and initiates the wrinkle removalmodule 132. If it is determined through the test module 134 that thereare no wrinkles remaining at 210, then the process returns to the basemodule 126 and the process ends.

Functioning of the fabric detection module 128 will now be explainedwith reference to FIG. 3. One skilled in the art will appreciate that,for this and other processes and methods disclosed herein, the functionsperformed in the processes and methods may be implemented in differingorder. Furthermore, the outlined steps and operations are only providedas examples, and some of the steps and operations may be optional,combined into fewer steps and operations, or expanded into additionalsteps and operations without detracting from the essence of thedisclosed embodiments.

The flow chart of FIG. 3 shows the architecture, functionality, andoperation of a possible implementation of the fabric detection module128. The process begins at 302 with the fabric detection module 128being initiated by, e.g., the base module 126, at 202 of FIG. 2. At 304,the sensor 120, vision device 124 or camera on the robotic system 102(FIG. 1), which may be a single camera or multiple cameras, can captureone or more images of at least a portion of the work table (or worksurface). There may be multiple images captured and the vision device124 or camera can be the RGB-D camera. An RGB-D camera is a digitalcamera that can provide color (RGB) and depth information for pixels ina captured image, at 304. In some implementations, images can becaptured at a fixed frequency (e.g., 10 Hz) or interval duringprocessing or handling of the piece of product.

Next, the fabric detection module 128 determines if the capturedimage(s) contain a piece of the product (or work piece) at 306. This canbe accomplished by continuously comparing the images to a template imageof the work table (or work surface) and, if there is any difference,then it can be determined that a piece of the product is contained inthe images. Other implementations can use image statistics or itsderivatives, which may be used independently or in combination. Forexample, a chan-vese algorithm for image segmentation can maximize theseparation between mean intensities of two regions (e.g., an object andbackground). The difference in color in the captured images and thetemplate image of the work table can be considered. For example, if thework table is white in color and the pieces of the product material areblue, then when the captured images contain any blue it may bedetermined that a piece of the product is contained in the images, at306. If it is determined at 306 that the images contain a piece of theproduct then the fabric detection module 128 returns at 308 to the basemodule 126 (FIG. 2), where the depth filtering can be initiated at 204.If it is determined that the captured images do not contain a piece ofthe product at 306, then the process can determine if there was an errorat 310. For example, if a piece of the product was scheduled fordelivery by a stacking device but is identified at 306, then an errorindication can be provided to, e.g., an operator for appropriate actionor the robotic system 102 can be stopped. If no error is present at 310,then the process returns to 304 where additional images can be capturedto detect the presence of product material on the work table. If thereis an error, the process can return to the base module 126 at 308.

Functioning of the depth filtering module 130 will now be explained withreference to FIG. 4. One skilled in the art will appreciate that, forthis and other processes and methods disclosed herein, the functionsperformed in the processes and methods may be implemented in differingorder. Furthermore, the outlined steps and operations are only providedas examples, and some of the steps and operations may be optional,combined into fewer steps and operations, or expanded into additionalsteps and operations without detracting from the essence of thedisclosed embodiments.

The flow chart of FIG. 4 shows the architecture, functionality, andoperation of a possible implementation of the depth filtering module130. The process begins at 402 with the depth filtering module 130 beinginitiated by, e.g., the base module 126, at 204 of FIG. 2. At 404, thedepth filtering module 130 performs an image rectification process,which is a transformation process used to project images on a commonplane. It can be used in geographic information systems to merge imagestaken from multiple perspectives into a common map coordinate system,which in this process may be used to determine if a piece of product islying flat on the work table or contains a wrinkle. The depth filteringmodule 130 can then determine the boundary of the product at 406. Theboundary of the product can be determined from color images using avariety of techniques such as, e.g., an image segmentation that caninclude a Gaussian Mixture Model, which allows the foreground of acaptured image to be extracted for further processing by removing thebackground. Some implementations can include the use of objectrecognition at 406 in order to determine the boundaries of the piece ofproduct.

Next, the depth filtering module 130 generates a depth map of the pieceof product (or work piece) at 408 using information from the capturedimages and/or other sensors. Generating the depth map can includeperforming an image registration process at 408, which overlays two ormore images from various sensors 120 such as imaging equipment or othersensors taken at different times and/or angles or from the same scene togeometrically align the images for analysis. The sensor(s) 120 used inthis process can be any one of, but not limited to, an RGB-D camera,time of flight camera, IP camera, light-field camera, monorail camera,multiplane camera, rapatronic camera, stereo camera, still camera,thermal imaging camera, acoustic camera, rangefinder camera, etc.

Various approaches can be used to generate the depth map. For example,stereo vision can use two optically identical cameras that arecalibrated and separated by a known distance along a baseline. The imageof the work piece on the work table can be acquired by the camerasconcurrently. The perspective projection of the piece of product on thetwo cameras produces a slightly shifted version of the same scene whichcan be used to determine 3D information.

Feature descriptors such as, e.g., scale invariant feature transform(SIFT) or histogram of gradients (HoG) can be used to select imagefeatures in the reference image. A disparity map can then be calculatedby finding the corresponding points for the feature descriptors in thematch image. The corresponding feature points can be found by using avariety of techniques, such as Sum of Squared Differences (SSD) orAbsolute intensity Differences (AD). The unconstrained search space canbe the entire second image, however this search space can be limited bytaking advantage of the epipolar geometry. This geometry describes theintrinsic projective geometry relationship between two views. The searchspace can thus be reduced to be a one dimensional epipolar line in thesecond image. The orientation of the epipolar line can be furtherconstrained by applying a homography to each image, such that thereprojected image planes are parallel to each other. This transformationensures that epipolar line is aligned along the horizontal axis of thesecond image to further reduce the computational burden. The search forthe feature descriptor is executed along this horizontal epipolar line.The difference in positions of the feature descriptor in the two imagesprovide the disparity metric for the feature descriptor. This isrepeated for all the feature descriptors fin the first image. The depthmap is inversely proportional to the disparity map and the constant ofproportionality can be calculated by triangulation.

Another approach to generate a depth map can use active stereo visionwhere one of the cameras of the stereo system is replaced by astructured light (an intense beam of coherent light such as, e.g., alaser) projector. For a calibrated projector, the correspondence problemcan be solved by searching for the projected line in the camera image(match image). This search can be further constrained by the fact thatthe order of corresponding points remain unchanged. The disparity mapcan be constructed by raster scanning the object in the scene with theprojector and building the disparity map one line at a time.

At 410, the depth filtering module 130 can remove any depth sensornoise, or any other undesired effects using, e.g., temporal and spatialimage filters. For example, the depth map can be preprocessed by aspatio-temporal filter operating simultaneously across multiple images(space and time) to mitigate the measurement noise. Any depth bias,which may arise from an uneven table, can be removed with model-fittingor look-up table approaches. In various implementations, the look-uptables can be calculated or estimated frame by frame in close toreal-time or the look-up tables may be precomputed in advance. In someimplementations, the boundary detection at 408 can be implemented afterthe depth map is generated in order to utilize information from thedepth map.

The depth map can then be processed at 412 to determine if the image ofthe piece of product contains a wrinkle or multiple wrinkles. As apreprocessing step, the imaging plane in the 3D scene (work tablesurface) can be transformed to be parallel to the image sensor plane,such that it its normal is aligned along the optical axis of the imagecamera, using a transformation (homography). This homography can becalculated from the known geometry and intrinsic parameters of theimaging system (camera) and 3D scene.

The depth map can be further used to model the object (piece of product)in the scene to be imaged can be modeled as a two dimensional roughsurface (a surface with large deviations in the local normal vector withrespect to it planar vector). A height distribution model can becalculated for the rough surface, and the height of surface can bemodeled as a two dimensional probability distribution. Prior knowledgeabout the surface properties of the piece of product being imaged can beused to refine the estimated height distribution model. First,statistical height descriptors can be calculated as a measure of averageroughness at a local scale, and spatial descriptors such as peakdensity, zero crossing density etc. can be calculated to model theheight distribution at a global (image) scale. A wrinkle can becharacterized by appreciable variations in both local and globaldescriptors, compared to those of a planar surface. The height andspatial descriptors can be processed (e.g., by a higher level decisionmaking algorithm) to identify the areas of the image containing awrinkle.

The surface map of the object offers another representation of the depthmap. The local curvatures can be calculated from the surface map. Thelocal curvatures describe the individual and joint curvature along itsaxes of the image. A wrinkle can be characterized by a ridge thattravels along the wrinkle and a curved shape in the other direction. Theprincipal axes of curvature can be calculated using eigenvector analysison the matrix describing the local curvature. The largest (of the two)eigenvector points in the direction that the fabric should be tensionedto remove the wrinkle. Tensioning fabric in that direction with fabricmover(s), or material mover(s), can dewrinkle the fabric.

The presence of a wrinkle can be evaluated based upon a wrinkle profilethat can include models, parameters, limits, thresholds, etc. related tovarious wrinkle characteristics such as, e.g., ridge height, ridgewidth, orientation, etc. that can specify acceptable conditions orcriteria for processing of the piece of the product. The wrinkle profilecan also account for which sensors 120 or vision devices 124 are used tocapture the images. If the product does contain a wrinkle at 414, thedepth filtering module 130 returns to the base module 126 (FIG. 2) at416, where wrinkle removal can be initiated at 206. If it is determinedthat no wrinkles are identified or remain in the product, then theprocess ends at 418.

Functioning of the wrinkle removal module 132 will now be explained withreference to FIG. 5. One skilled in the art will appreciate that, forthis and other processes and methods disclosed herein, the functionsperformed in the processes and methods may be implemented in differingorder. Furthermore, the outlined steps and operations are only providedas examples, and some of the steps and operations may be optional,combined into fewer steps and operations, or expanded into additionalsteps and operations without detracting from the essence of thedisclosed embodiments.

The flow chart of FIG. 5 shows the architecture, functionality, andoperation of a possible implementation of the wrinkle removal module132. The process begins at 502 with the wrinkle removal module 132 beinginitiated by, e.g., the base module 126, at 206 of FIG. 2. Then at 504the wrinkle removal module 132 determines the budgers (or other fabricmover(s), or material mover(s), 116 of FIG. 1) needed to remove thewrinkle or wrinkles. This can be accomplished by the robotic system 102knowing the location of all the budgers on the work table and thelocation of the product on the work table (or surface). An algorithm candetermine which budgers or other fabric mover(s) 116 surrounding thelocation of the wrinkle on the product can be used to straighten out thewrinkle and what corrective actions are needed to remove the wrinkle, at504. The product can be manipulated to adjust multiple wrinkles at thesame time. At 506, the wrinkle removal module 132 initiates activationof the identified budgers or fabric mover(s) 116 to remove the wrinklefrom the product material. For example, the activated budgers can beinitiated or controlled to turn, rotate, or move in a specific directionto reduce or remove the wrinkle. In some implementations, budgers maynot be the only actuator used. For example, the robotic system 102 coulduse a mechanical arm, clamp, and/or various other types of endeffectors, or some combination of these, in order to reduce or removethe wrinkles.

It should be noted, that the role of a budger system is tosimultaneously move and feed product parts to other robotic systemsusing a set of distributed actuators. Each budger can move the productin contact with that budger independently in a given direction and/or ata given speed. Wrinkles and/or folds may be introduced while the productis moving on the budger system. Those budgers may also be used tocorrect wrinkles and folds through differential movement of the budgers.In addition, the budgers may be used in the sewing process bymanipulating the products position while it is being sewn. Next, thevision device(s) 124 or camera captures one or more additional images ofthe piece of product at 508. In some implementations, images can becaptured at a fixed frequency (e.g., 10 Hz) or interval duringprocessing or handling of the piece of product. Then at 510, the wrinkleremoval module 132 returns to the base module 126 (FIG. 2), wheretesting to confirm that the wrinkle has been reduced or removed can beinitiated at 208.

Functioning of the test module 134 will now be explained with referenceto FIG. 6. One skilled in the art will appreciate that, for this andother processes and methods disclosed herein, the functions performed inthe processes and methods may be implemented in differing order.Furthermore, the outlined steps and operations are only provided asexamples, and some of the steps and operations may be optional, combinedinto fewer steps and operations, or expanded into additional steps andoperations without detracting from the essence of the disclosedembodiments.

The flow chart of FIG. 6 shows the architecture, functionality, andoperation of a possible implementation of the test module 134. Theprocess begins at 602 when the test module 134 is initiated by, e.g.,the base module 126, at 208 of FIG. 2. The test module 134 then performsan image rectification process at 604, which is a transformation processused to project images on a common plane. It can be used in geographicinformation systems to merge images taken from multiple perspectivesinto a common map coordinate system, which in this process may be usedto determine if a piece of product is lying flat on the work table orcontains a wrinkle. The test module 134 can next determine the boundaryof the product at 606. The boundary of the product can be determinedfrom color images using a variety of techniques such as, e.g., an imagesegmentation that can include a Gaussian Mixture Model, which allows theforeground of a captured image to be extracted for further processing byremoving the background. Some implementations can include the use ofobject recognition at 606 in order to determine the boundaries of thepiece of product.

Next, the test module 134 generates a depth map of the piece of product(or work piece) at 608 using information from the additional capturedimage(s). Generating the depth map can include performing an imageregistration process at 608, which overlays two or more images fromvarious sensors 120 and/or vision devices 124 such as imaging equipmentor sensors taken at different times and/or angles or from the same sceneto geometrically align the images for analysis. The vision device(s) 124used in this process can be any one of, but not limited to, an RGB-Dcamera, time of flight camera, IP camera, light-field camera, monorailcamera, multiplane camera, rapatronic camera, stereo camera, stillcamera, thermal imaging camera, acoustic camera, rangefinder camera,etc.

At 610, the test module 134 can remove any depth sensor noise, or anyother undesired effects using, e.g., temporal and spatial filters. Anydepth bias, which may arise from an uneven table, can be removed withmodel-fitting or look-up table approaches. In various implementations,the look-up tables can be calculated or estimated frame by frame inclose to real-time or the look-up tables may be precomputed in advance.In some implementations, the boundary detection at 608 can beimplemented after the depth map is generated in order to utilizeinformation from the depth map.

The depth map can then be processed at 612 to determine if the piece ofproduct still contains a wrinkle. The presence of the wrinkle can beevaluated based upon the wrinkle profile for processing of the piece ofthe product. If the product does contain a wrinkle at 614, the testmodule 134 returns to the base module 126 (FIG. 2) at 616, where wrinkleremoval can be initiated at 206. If it is determined that no wrinklesare identified or remain in the product, then the process ends at 618.

Functioning of the wrinkle removal process will now be discussed withreference to the captured images of FIGS. 7A and 7B. One skilled in theart will appreciate that, for this and other processes and methodsdisclosed herein, the functions performed in the processes and methodsmay be implemented in differing order. Furthermore, the outlined stepsand operations are only provided as examples, and some of the steps andoperations may be optional, combined into fewer steps and operations, orexpanded into additional steps and operations without detracting fromthe essence of the disclosed embodiments.

FIGS. 7A and 7B display captured images that illustrate a wrinkleremoval example. FIG. 7A of the wrinkle removal example displays animage of a piece of product that has been processed through the depthfiltering module 130 or the test module 134. The lines located withinthe piece of product represent wrinkles, or peaks or bumps that havebeen identified on the product. This is the result of the analysis ofthe depth filtering module 130 (or the test module 134) that allows itto determine if there are wrinkles on the product, identify where thosewrinkles are in relation to the budgers (or other fabric mover(s) 116)about the work table and the product, and the direction in which thosebudgers (or other fabric mover(s) 116) need to move in order to removethe wrinkle(s). FIG. 7B of the wrinkle removal example displays an imageof a piece of product that does not contain any wrinkles. This image canbe analyzed through the depth filtering module 130 or the test module134, and since there are no wrinkles detected the process would endallowing the piece of product to continue being processed on the worktable.

Referring now to FIG. 8, functioning of an example of the endeffector(s) will now be explained. One skilled in the art willappreciate that, for this and other processes and methods disclosedherein, the functions performed in the processes and methods may beimplemented in differing order. Furthermore, the outlined steps andoperations are only provided as examples, and some of the steps andoperations may be optional, combined into fewer steps and operations, orexpanded into additional steps and operations without detracting fromthe essence of the disclosed embodiments.

FIG. 8 shows an example of a matrix of various end effectors that can beused with the robotic system 102 of FIG. 1. The table illustrates thatnot only budgers may be used to move and alter the product in order toremove the identified wrinkles, but this can also be accomplishedthrough mechanical arms, clamps, or end effector N (representing anumber of apparatuses or tools that may be used or a combination thatmay be used to move or alter the product). Multiple budgers may be usedto alter the product, however a combination of budgers and mechanicalarms, budgers and clamps, or budgers and end effector N may be used inorder to alter and/or move the product to remove the wrinkles. Also,multiple mechanical arms or a combination of mechanical arms and clamps,or mechanical arms and end effector N may be used to move or alter theproduct. Additionally, multiple clamps (attached to some sort ofactuator) or a combination of clamps and end effector N may be used toalter or move the product.

It should be emphasized that the above-described embodiments of thepresent disclosure are merely possible examples of implementations setforth for a clear understanding of the principles of the disclosure.Many variations and modifications may be made to the above-describedembodiment(s) without departing substantially from the spirit andprinciples of the disclosure. All such modifications and variations areintended to be included herein within the scope of this disclosure andprotected by the following claims.

The term “substantially” is meant to permit deviations from thedescriptive term that don't negatively impact the intended purpose.Descriptive terms are implicitly understood to be modified by the wordsubstantially, even if the term is not explicitly modified by the wordsubstantially.

It should be noted that ratios, concentrations, amounts, and othernumerical data may be expressed herein in a range format. It is to beunderstood that such a range format is used for convenience and brevity,and thus, should be interpreted in a flexible manner to include not onlythe numerical values explicitly recited as the limits of the range, butalso to include all the individual numerical values or sub-rangesencompassed within that range as if each numerical value and sub-rangeis explicitly recited. To illustrate, a concentration range of “about0.1% to about 5%” should be interpreted to include not only theexplicitly recited concentration of about 0.1 wt % to about 5 wt %, butalso include individual concentrations (e.g., 1%, 2%, 3%, and 4%) andthe sub-ranges (e.g., 0.5%, 1.1%, 2.2%, 3.3%, and 4.4%) within theindicated range. The term “about” can include traditional roundingaccording to significant figures of numerical values. In addition, thephrase “about ‘x’ to ‘y’” includes “about ‘x’ to about ‘y’”.

1. A robotic system, comprising: at least one material mover; a basemodule for wrinkle removal; and processing circuitry comprising aprocessor, wherein execution of the base module causes the roboticsystem to: receive a piece of product on a work surface; and adjust thepiece of product on the work surface using the at least one materialmover to reduce a wrinkle in the piece of product on the work surface.2. The robotic system of claim 1, comprising a vision device configuredto image the work surface, wherein execution of the base module causesthe robotic system to further determine whether the wrinkle afteradjustment satisfies a wrinkle profile based upon analysis of one ormore images obtained by the vision device of the robotic system.
 3. Therobotic system of claim 2, comprising in response to the wrinkleexceeding the wrinkle profile after adjustment, execution of the basemodule causes the robotic system to further adjust the piece of producton the work surface to further reduce the wrinkle.
 4. The robotic systemof claim 1, wherein the robotic system adjusts the piece of product onthe work surface to reduce the wrinkle in response to the piece ofproduct containing the wrinkle exceeding a wrinkle profile.
 5. Therobotic system of claim 4, comprising: obtaining, by a sensor, one ormore images of the piece of product; and execution of a base moduledetermines whether the wrinkle satisfies the wrinkle profile based uponanalysis of the one or more images.
 6. The robotic system of claim 1,wherein presence of the piece of product is determined based uponevaluation of one or more images obtained by a sensor of the roboticsystem.
 7. The robotic system of claim 1, wherein adjustment of thepiece of product on the work surface is in response to the wrinkleexceeding a wrinkle profile.
 8. The robotic system of claim 7, whereinexecution of the base module determines whether the piece of productcontains a wrinkle based upon analysis of one or more images withrespect to the wrinkle profile.
 9. The robotic system of claim 1,wherein the at least one material mover comprises a blower configured toprovide directed air jets.
 10. The robotic system of claim 9, whereinthe air jets are directed across the piece of product substantiallyperpendicular to a primary axis of the wrinkle.
 11. The robotic systemof claim 1, wherein the at least one material mover secures an edge ornear an edge of the piece of product during adjustment.
 12. A method forremoval of wrinkles in a piece of product by a robotic system,comprising: receiving a piece of product; positioning the piece ofproduct on a work surface; and adjusting the piece of product on thework surface using a fabric mover of the robotic system, the adjustmentreducing a wrinkle in the piece of product on the work surface.
 13. Themethod of claim 12, comprising identifying whether the piece of producton the work surface contains the wrinkle based upon one or more imagesobtained by a vision device of the robotic system and a wrinkle profile,wherein the piece of product is adjusted in response to the wrinklefailing to satisfy the wrinkle profile.
 14. The method of claim 13,wherein identifying whether the piece of product contains the wrinklecomprises generating a depth map from of the one or more images.
 15. Themethod of claim 12, comprising determining whether the wrinkle afteradjustment satisfies a wrinkle profile based upon one or more imagesobtained by a vision device.
 16. The method of claim 15, comprisingfurther adjusting the piece of product on the work surface in responseto failing to satisfy the wrinkle profile to further reduce the wrinkle.17. The method of claim 12, comprising determining a boundary of thepiece of product on the work surface from based upon one or more imagesobtained by a vision device of the robotic system.
 18. The method ofclaim 12, wherein the adjustment reducing a wrinkle in the piece ofproduct comprises providing directed air jets with respect to a surfaceof the piece of product.
 19. The method of claim 18, wherein the airjets are directed across the piece of product substantiallyperpendicular to a primary axis of the wrinkle.
 20. The method of claim12, wherein the piece of product is adjusted during movement on the worksurface.